Method and system for representing and interacting with augmented reality content

ABSTRACT

Systems and methods for displaying augmented reality (AR) content are disclosed. The AR device may include a display configured to display real-world content overlaid with AR content and at least one sensor configured to provide an output indicative of an orientation, location, or motion of the AR device. The system may also include a processor device configured to: cause the AR content to be shown on the display at an initial location on the display; determine a change in orientation of the AR device based on the output of the at least one sensor; and change a position of the AR content on the display to a second location on the display, wherein the change in position of the AR content from the initial location to the second location is related to the determined change in orientation of the AR device.

PRIORITY

This application is based on and claims priority to U.S. ProvisionalApplication No. 61/798,846, filed on Mar. 15, 2013, which isincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to augmented reality contentand more particularly, to methods and systems for representing andinteracting with augmented reality content.

BACKGROUND

Technology advances have enabled mobile personal computing devices tobecome more capable and ubiquitous. In many cases these devices willhave both a display as well as a combination of sensors, for example,GPS, accelerometers, gyroscopes, cameras, light meters, and compasses orsome combination thereof. These devices may include mobile computingdevices as well as head mounted displays.

These mobile personal computing devices are increasingly capable of bothdisplaying information for the user as well as supplying contextualinformation to other systems and applications on the device. Suchcontextual information can be used to determine the location,orientation and movement of the user interface display of the device.

SUMMARY

In some embodiments, an augmented reality (AR) device is provided. TheAR device may include a see-through or semi-transparent display (e.g., adisplay that allows transmission of at least some visible light thatimpinges upon the AR device) that allows the user to see the real-worldenvironment and to display generated images superimposed over orprovided in conjunction with a real-world view as perceived by thewearer through the lens elements. The AR device may also includeelectronic or analog sensors that can be used to establish and/or trackthe physical context of the display. By way of example and withoutlimitation, the sensors could include any one or more of a motiondetector (e.g., a gyroscope and/or an accelerometer), a camera, alocation determination device (e.g., a GPS device, an NFC reader), amagnetometer, and/or an orientation sensor (e.g., a theodolite,infra-red sensor).

In some embodiments, the display on the AR device may provide a visualrepresentation of some number of virtual objects in two-dimensional orthree-dimensional space that are visible in the user's current field ofview. These virtual objects make up the virtual content displayed on theAR device. The virtual content may overlay the real world contentvisible to the user, creating an Augmented Reality (AR) environment forthe user. Additionally, the display may include a visual representationof a reticle with a fixed point of reference to the user, or othervirtual selection objects or pointers that allow the user to selectspecific virtual objects.

A user wishing to move an AR object within or out of the field of viewmay use the reticle or other selection device to select the AR content.After selecting the AR content, the user can then designate a positionin which to place the AR content. The position can be anywhere insidethe field of view or anywhere outside of the field of view of the ARdevice. If the AR content is placed outside of the field of view of theuser, then the content will not be visible to the user until the fieldof view of the AR device is moved so that it overlaps with theuser-designated position of the AR content. When the AR content isreturned to the field of view of the AR device, the display of the ARcontent may then track the movement of the field of view of the ARdevice according to user input.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary system for implementing embodimentsconsistent with the disclosure;

FIG. 2 illustrates an exemplary AR device;

FIG. 3 is a block diagram illustrating a configuration of asoftware-driven system consistent with the exemplary disclosedembodiments;

FIG. 4 illustrates an initial position of the AR device and a perceivedlocation of AR content in the respective coordinate systems consistentwith the exemplary disclosed embodiments;

FIG. 5 illustrates an example of snapped AR content in a coordinatesystem consistent with the exemplary disclosed embodiments;

FIG. 6 illustrates an example of AR content movement consistent with theexemplary disclosed embodiments;

FIG. 7 illustrates another example of AR content movement consistentwith the exemplary disclosed embodiments;

FIG. 8 illustrates an example of updating a display of AR contentconsistent with the exemplary disclosed embodiments; and

FIG. 9 illustrates another example of updating a display of AR contentconsistent with the exemplary disclosed embodiments.

DETAILED DESCRIPTION

Mobile personal computing devices may include one or more portabledisplays used to overlay virtual objects with real world objects.Virtual content that relates to one or more real world objects (e.g.,places, things, people, etc.) and that may be provided on a display maybe referred to as Augmented Reality (AR) content. Such AR content may beprovided on a display together with the real world objects to which theAR content relates. Further, the views of the real world objects on adisplay may correspond to computer-generated representations of thoseobjects or, in some embodiments (e.g., where at least a portion of thedisplay passes or channels light), may correspond to actual,non-computer-generated views of the objects.

In many instances, displaying the AR content can interfere with the viewof the real world. For example, displayed AR content may overlap with anobject of interest in the real world and may obscure either partially orfully a user's view of the real world object of interest. This can beinconvenient, distracting, or dangerous to the user. In other instances,an amount of available AR content may exceed the display capabilities ofthe device or may require more space on the display than is availablewithin a designated or predetermined virtual content display area on thedisplay. Some disclosed embodiments may address these issues by enablinga user to reposition AR content on a display such that it does notinterfere with (or at least interferes with less of) the user's view ofreal world objects. The disclosed embodiments may also facilitatesubsequent and repeated accessing of repositioned AR content.

A method and system for enabling users to select, position, store andreference virtual content within an augmented reality context will inmany cases be more efficient, more intuitive, and safer than usingalgorithms to scale or move content within the view, or other similartechniques.

Exemplary methods and systems are described herein. It should beunderstood that the word “exemplary” is used herein to mean “serving asan example, instance, or illustration.” Any embodiment or featuredescribed herein as “exemplary” is not necessarily to be construed aspreferred or advantageous over other embodiments or features. Theexemplary embodiments described herein are not meant to be limiting. Itwill be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

FIG. 1 illustrates an exemplary system 100 for implementing embodimentsconsistent with disclosed embodiments. In one aspect, system 100 mayinclude a server system 110, a user system 120, and network 130. Itshould be noted that although a single user system 120 is shown in FIG.1, more than one user system 120 may exist in system 100. Furthermore,although a single server system 110 is shown in FIG. 1, more than oneserver system 110 may exist in system 100.

Server system 110 may be a system configured to provide and/or manageservices associated with providing AR content to users. Consistent withthe disclosure, server system 110 may provide AR content to user system120 based on information received from user system 120. For example,server system 110 may provide AR content to user system 120 based on adetected position, detected position change, or detected/determinedcontext of user system 120. Such AR content may also be provided to usersystem 120 based upon requests received from user system 120 or based ona detected and/or recognized object within a field of view of an imagingdevice associated with user system 120. In general, any type of ARcontent, including any desired form of information, can be provided fromserver system 110 to user system 120 based on the requirements of aparticular application. Further, any suitable trigger may be used toinitiate transfer and/or updating of AR content provided to user system120 by server system 110.

Server system 110 may include one or more components that performprocesses consistent with the disclosed embodiments. For example, serversystem 110 may include one or more computers, e.g., processor device111, a CPU, and/or other hardware-based applications processing devices,configured to execute software instructions programmed to performaspects of the disclosed embodiments. Such software instructions may bestored, for example, in database 113 and when implemented may createand/or maintain a global coordinate system, provide AR objects to usersystems for display, transmit information associated with the AR objectsto user system 120, or any other relevant function. In one aspect,database 113 may be co-located with server system 110. Alternatively,database 113 may be located remotely from the server system 110.Database 113 may include computing components (e.g., database managementsystem, database server, etc.) configured to receive and processrequests for data stored in memory devices of database(s) 113 and toprovide data from database 113.

User system 120 may include a system associated with a user (e.g., aconsumer, field technician, equipment operator, or any other individualthat may benefit from received AR content) that is configured to performone or more operations consistent with the disclosed embodiments. In oneembodiment, a user may operate user system 120 to perform one or moresuch operations. User system 120 may include a communication interface121, a processor device 123, a memory 124, a sensor array 125, a display122, and/or any other components that may facilitate the display of ARcontent to the user. The processor device 123 may be configured toexecute software instructions to perform aspects of the disclosedembodiments. User system 120 may be configured in the form of an ARdevice, such as a head mounted display (HMD). Although in the presentdisclosure user system 120 is described in connection with a HMD, usersystem 120 may include tablet devices, mobile phone(s), laptopcomputers, and any other computing device(s) known to those skilled inthe art.

Communication interface 121 may include one or more communicationcomponents, such as cellular, WIFI, Bluetooth transceivers, or any otherwireless transceivers or communication equipment. Communicationinterface 121 may package and send user commands or input across network130 to server system 110. Based on these commands and/or input, serversystem may return additional or updated AR content to be displayed tothe user by user system 120. This additional or updated AR content, orinstructions for providing the additional or updated AR content, may bereceived from server system 110 via communication interface 121.Processor device 123 may access and use information received viacommunication interface 121.

The display 122 may include any suitable display for providing ARcontent to a user. For example, in some embodiments, display 122 mayinclude one or more translucent, transparent, or semi-transparentcomponents configured to pass at least some light from a scene to theeyes of a user. The display 122 may also include opaque lenses orcomponents, e.g., where the images seen by the user arecomputer-generated, constitute light-guided projections of real worldobjects, or include images captured by one or more cameras (or otherimage acquisition devices) associated with user system 120. In suchembodiments, generated, captured, and/or light-guided images of realworld objects may be projected onto opaque or semi-opaque componentsassociated with display 122. Display 122 may also project informationusing holographic images.

The sensor array 125 may include any type of sensor configured toprovide an output signal useful in providing AR content to a user. Insome embodiments, sensor array 125 may include one or more GPS sensors,cameras, barometric sensors, proximity sensors, physiological monitoringsensors, chemical sensors, magnetometers, gyroscopes, accelerometers,bar code scanners, motion detectors, etc.

Processor devices 111 and 123 may include one or more suitablehardware-based processing devices, such as a microprocessor, controller,central processing unit, etc. that can execute software-basedinstructions. In some embodiments, processor devices 111 and/or 123 mayinclude a microprocessor from the OMAP family manufactured by TexasInstruments, the Snapdragon™ family manufactured by Qualcomm™, or any ofvarious types of processor devices manufactured by other microprocessormanufacturers.

Consistent with disclosed embodiments, one or more components of system100, including server system 110 and user system 120, may also includeone or more memory devices (such as memories 112 and 124) as shown inexemplary form in FIG. 1. The memory devices may include magneticstorage units, optical storage units, RAM, ROM, hard drives, flashmemory, etc. Such memory units may be located together with serversystem 110 and/or user system 120 or may be located remotely withrespect to either or both systems. Memory 112 and or 124 may storesoftware instructions that are executed by processor devices 111 and123, such as one or more applications, network communication processes,operating system software, software instructions relating to thedisclosed embodiments, and any other type of application or softwareknown to be executable by processing devices. The memory devices may bea volatile or non-volatile, magnetic, semiconductor, tape, optical,removable, non-removable, or other type of storage device ornon-transitory computer-readable medium. The memory devices may be twoor more memory devices distributed over a local or wide area network, ormay be a single memory device. In certain embodiments, the memorydevices may include database systems, such as database storage devices,including one or more database processing devices configured to receiveinstructions to access, process, and send information stored in thestorage devices. By way of example, database systems may includingOracle™ databases, Sybase™ databases, or other relational databases ornon-relational databases, such as Hadoop sequence files, HBase, orCassandra.

In some embodiments, server system 110 and user system 120 may alsoinclude one or more additional components (not shown) that providecommunications with other components of system environment 100, such asthrough network 130, or any other suitable communicationsinfrastructure.

Network 130 may be any type of network that facilitates communicationsand data transfer between components of system environment 100, such as,for example, server system 110 and user system 120. Network 130 may be aLocal Area Network (LAN), a Wide Area Network (WAN), such as theInternet, and may include a single network or a combination of networks.Further, network 130 may include a single type of network or acombination of different types of networks, such as the Internet andpublic exchange networks for wireline and/or wireless communications.Network 130 may utilize cloud computing technologies that are familiarin the marketplace. Network 130 is not limited to the above examples,and system 100 may implement any type of network that allows theentities (and others not shown) included in FIG. 1 to exchange dataand/or information.

FIG. 2 illustrates an exemplary AR device 200, which may correspond toand include any of the components and features of user system 120. Atpresent, augmented reality glasses are the most common type of ARdevice. It is also possible to use a smart phone as an AR device.Therefore, it will be understood that the present invention is notlimited to augmented reality glasses or any one type of AR device. Forexample, a relatively simple AR device might involve a projector with acamera interacting with the surrounding environment, where theprojection could be on a glass surface or on top of other objects.

As shown in FIG. 2, the AR glasses 200 may include features relating tonavigation, orientation, location, sensory input, sensory output,communication and computing. For example, the AR glasses 200 may includean inertial measurement unit (IMU) 201. IMU 201 may include axialaccelerometers or gyroscopes for measuring position, velocity andorientation. IMUs may enable determination of the position, velocityand/or orientation of the AR glasses within the surrounding real worldenvironment and/or its position, velocity and orientation relative toreal world objects within that environment in order to perform itsvarious functions.

The AR glasses 200 may also include a Global Positioning System (GPS)unit 202. GPS units receive signals transmitted by a plurality ofgeosynchronous earth orbiting satellites in order to triangulate thelocation of the GPS unit. In more sophisticated systems, the GPS unitmay repeatedly forward a location signal to an IMU to supplement theIMUs ability to compute position and velocity, thereby improving theaccuracy of the IMU. In the present case, the AR glasses 200 may employGPS to identify a location of the AR glasses.

As mentioned above, the AR glasses 200 may include a number of featuresrelating to sensory input and sensory output. AR glasses 200 may includeat least a front facing camera 203 to provide visual (e.g., video)input, a display (e.g., a translucent or a stereoscopic translucentdisplay) 204 to provide a medium for displaying computer-generatedinformation to the user, a microphone 205 to provide sound input andaudio buds/speakers 206 to provide sound output. In some embodiments,the visually conveyed digital data may be received by the AR glasses 200through the front facing camera 203.

The AR glasses 200 may also have communication capabilities, similar toother mobile devices, through the use of a cellular, WiFi, Bluetooth ortethered Ethernet connection. The AR glasses 200 may also include anon-board microprocessor 208. The on-board microprocessor 208 may controlthe aforementioned and other features associated with the AR glasses200.

In one exemplary embodiment, the AR glasses 200 may include asee-through display and sensor systems that provide the device'slocation, orientation, and bearing (for example, latitude, longitude,altitude, pitch, roll or degree tilt from horizontal and vertical axes,and compass heading). The AR glasses 200 could be configured as glassesthat can be worn by a person. Further, one or more elements of thesensor system may be located on peripheral devices physically separatefrom the display.

Additionally, in this embodiment, the AR glasses 200 may rely on acomputer software application to instruct the glasses to render virtualobjects on the display field of view. Virtual objects include, but arenot limited to, text, images, models, icons. The user may view orinteract with virtual objects using the hardware and softwareapplication associated with the AR glasses 200.

FIG. 3 is a system block diagram 300 illustrating a software-drivensystem associated with an AR device, in accordance with exemplaryembodiments of the present invention. As illustrated, the softwaresystem may be configured into three layers. At the lowest layer is theaugmented reality device operating system 60. The operating system 60may, for example, be an Android based operating system, an iPhone basedoperating system, a Windows Mobile operating system or the like. At thehighest layer may reside a third party applications layer 62.Applications that are designed to work with the operating system 60 thateither came with the AR device or were loaded by the user may reside inthis third layer.

The middle layer is referred to as the augmented reality shell 64. Theaugmented reality shell 64, as shown, includes a number of componentsincluding a command processor 66, an environmental processor 68, arendering services module 70, and a network interaction services module72. It should be understood that each of the functional modules and/orcomponents may be hardware, software, firmware or a combination thereof.

The environmental processor 68, in general, monitors the surrounding,real world environment of the AR device based on input signals receivedand processed by the AR device (e.g., from sensor array 125). Theenvironmental processor 68 may be implemented, as shown in FIG. 3,similar to the other processing components, or it may be implementedseparately, for example, in the form of an application specificintegrated chip (ASIC). In one embodiment, the environmental processor68 may be configured to operate whenever the AR device is powered on.

The environmental processor 68, in turn, may also include severalprocessing modules: a visual processing module 74, a geolocationalprocessing module 76, and a positional processing module 78. The visualprocessing module 74 is primarily responsible for processing thereceived video, detecting and decoding the frames and processing thedigital data included with the video that was visually conveyed to theAR device.

The geolocational module 76 receives and processes signals relating tothe location of the AR device. The signals may, for example, relate toGPS coordinates, the location of a WiFi hotspot, or the proximity to oneor more local cell towers.

The positional module 78 receives and processes signals relating to theposition, velocity, acceleration, direction and orientation of the ARdevice. The positional module 78 may receive these signals from an IMU(e.g., IMU 201). The positional processing module 78 may, alternativelyor additionally, receive signals from a GPS receiver, where it isunderstood that the GPS receiver can only approximate position (andtherefore velocity and acceleration) and where the positional processingmodule 78 can then provide a level of detail or accuracy based on theGPS approximated position. Thus, for example, the GPS receiver may beable to provide the general GPS coordinates of a movie theater, but thepositional processing module 78 may be able to provide the user'sorientation within the movie theater. The positional processing module80 may be employed in conjunction with the visual processing module 74to synchronize user head movements with viewing experiences (e.g., whatthe rendering services module 70 will render on the display and,therefore, what the user sees).

In addition to the environmental processor 68, the augmented realityshell 64 includes a command processor 66 and a rendering services module70. The command processor 66 processes messaging between the modulesand/or components. For example, after the visual processing module 74processes the digital data that was visually received through the video,the visual processing module 74 communicates with the command processor66 which, in turn, generates one or more commands to the renderingservices module 70 to produce the computer generated data (e.g., text,graphics, additional video, sound) that will be used to supplement thevideo and enhance the user's viewing experience.

The rendering services module 70 may process the content of the digitaldata that was visually received and, based on instructions providedthrough the command processor 66, generate and present (e.g., display)data in the form of sound, graphics/animation, text, additional videoand the like. The user can thus view the video and, in addition,experience the computer-generated information to supplement the videoand enhance the viewing experience.

In some embodiments, the software applications associated with the ARdevice may create and maintain coordinate systems that correspond to theposition and orientation of the AR device, the field of view of the ARdevice, and the virtual content. These coordinate systems may correspondto locations in the real physical world or may be referenced withrespect to the AR device or to any other suitable reference. Themaintained coordinate systems may include a two dimensional Cartesiancoordinate system, a three dimensional Cartesian coordinate system, atwo dimensional Spherical coordinate system, a three dimensionalSpherical coordinate system, or any other suitable coordinate system.

AR content, including virtual objects or information, may be rendered atpositions on the display of the AR device defined by the softwareapplication. These positions may be relative to the user. For example,the initial position for display may be in the center of the user'sfield of view or at any other desired location with the user's field ofview. The AR content may also be rendered such that the user perceivesthe AR content as associated with a particular location or object in thereal world. For example, the virtual content may be rendered on thedisplay to appear to the user as located at coordinate positionscorresponding to a location in the real-world, for example, referring toa specific geographic coordinate, or to the current coordinate locationof another person or object. User movements relative to the specificgeographic coordinate, person, object, etc. may prompt the system toupdate the display of the AR content such that it appears snapped to thespecific geographic coordinate, person, object, etc.

If the user desires to move or “snap” the virtual content to a newposition, the user may select a virtual object, or group of objects,using an input method supported by the hardware and software applicationassociated with the AR device. For illustration purposes, one methodcould be to move a selection device, such as a mouse or pointer device.Another method would be to use gesture inputs (pointing of a finger,etc.), a push button device, touch sensitive surface, or voice commandsto augment selection. Another exemplary method would be to use a reticlerendered on the display that would allow head-movements made by the userto indicate the direction or location of the virtual object to beselected. Following the selection, the user may define a new positionfor the virtual object relative to the display of the AR device.

For example, in an exemplary embodiment, a user may of user system120/AR glasses 200 may include a field technician, and the AR content tobe displayed may include a service manual for a particular type ofequipment. Initially, and by operation of a voice command, input device,etc., the technician may request display of AR content in the form ofthe service manual. In some embodiments, this information may beretrieved from server system 110 or may be retrieved from a local memory124 and may be displayed in display 122. Initially, the AR content maybe displayed in the center of the user's field of view. Such positioningwould enable natural access and viewing of the desired content. ARcontent in the center of the user's field of view, however, may bedistracting or may impede the view of the real world objects with whichthe technician needs to interact. Therefore, system 120 may beconfigured to re-position the AR content to another location on display122 or even to another location in virtual space (maintained in memory)relative to the user (e.g., relative to the user's head position). Aswill be described in detail below, the repositioned AR content can beaccessed by the user. For example, the user may turn his head in thedirection of the repositioned AR content and, based on the sensed motionof the head, the rendering of the AR content may be updated relative tothe motion of the user's head. In some embodiments, the rendered ARcontent may appear to the user as fixed at the repositioned location,and the user may turn his head in the direction of the repositionedlocation in order to bring the AR content toward and into the center ofthe field of view of display 122. Subsequent turning of the head awayfrom the repositioned location will cause the rendered AR content on thedisplay to move away from the center of the field of view. In this way,the user (e.g., the technician) may move the desired AR content to avirtual location that does not impede viewing of objects in the centerof the field of view, but that still allows access to the desired ARcontent, for example, by turning toward the virtual position of thatcontent.

The user may define the new position of the virtual object by using aninput device or method supported by the hardware and software associatedwith AR device. For example, one method would be to “drag and drop” byfirst selecting the content to be moved, moving it by dragging it to thedesired new location and deselecting it to drop it in the new,repositioned location. Establishing a new position of the virtual objectmay create a new coordinate or change an existing coordinate associatedwith the virtual object in the software application of the AR device.

The user (or routines in the software) may define additional snappingproperties or rules associated with the virtual objects or newcoordinates. These properties may include parameters, tags, orinstructions for the software application of the AR device. For example,a property of the “snapped” virtual AR content may be to share the newcoordinates for the object with other users of the system 100 Anotherexample may be to delete the new coordinates of the virtual objectsafter a period of time and then revert to the previous position. Yetanother example would be to lock the virtual object from being moved toanother coordinate by a different user of the AR device.

When the user initiates the steps to “snap” a virtual object to a newposition, the software application of the AR device may use informationfrom sensor systems to define a first coordinate system for the ARdevice and a second coordinate system for the virtual object that isselected by the user, which may be referred to as cardinal snappingherein.

FIG. 4 is a diagrammatic representation of a coordinate space 400consistent with the exemplary disclosed embodiments. As shown in FIG. 4,the orientation of the AR device 401 may be defined according to a firstcoordinate system (x, y, z), and the position (or perceived position) ofthe AR content 402 may be defined according to a second coordinatesystem (x′, y′, z′). In some embodiments, the origin of the secondcoordinate system may be defined by user input, such as through a menufunction, speech command, and so on. In some embodiments, the first andthe second coordinate systems may have the same origin when they aredefined. For example, in FIG. 4, the first and the second coordinatesystems have the same origin defined as (0, 0, 0) and (0′, 0′, 0′)respectively.

In some embodiments, the origin of the first coordinate system may becentered at the AR device, and the field of view of the user may betracked in the first coordinate system. In one embodiment, the field ofview may be associated with a vector having an origin at the position ofthe AR device in the first coordinate system and extending along a linenormal to the display of the AR device at the center of the field ofview of the AR device. For example, in FIG. 4, the user's field of viewmay be associated and tracked with a vector, v, having origin at (0, 0,5) and proceeding along the y-axis. Alternatively, the field of view canbe defined as a two-dimensional shape (e.g. an arc), or athree-dimensional shape (e.g. a conic section). The virtual object maybe given coordinates in the second coordinate system that associate itslocation relative to the current field of view of the user. For example,in FIG. 4, the coordinates of the AR content 402 are defined to be (0′,5′, 5′) in the second coordinate system. While the AR content may bedisplayed on a display associated with AR device 401, the user mayperceive the location of the AR content as residing at (0′, 5′, 5′) inthe second coordinate system.

When the user of the AR device moves, the AR device moves in acorresponding fashion with the user. For example, if the user translatesin space, the AR device will similarly translate, and if the user's headrotates, the AR device and the associated field of view vector willsimilarly change direction along with the head rotation. The sensors inthe AR device provide data to the application of AR device which mayprompt or enable the application to monitor information associated withthe display including, for example, the current location, orientationand/or bearing of the display unit. This information, in turn, may beused to update or change aspects of images or information presented tothe user within the user's field of view on the display unit,

The field of view of the user may intersect the virtual object. The usermay select the virtual object using input methods supported by thehardware and software of the AR device. The user may take action todefine a new position for the virtual object in the second coordinatesystem. The new position of the virtual object may be outside of thefield of view of the user. For example, the user may choose to move theAR content 90 degrees (or more) to the right of the user. In anotherexample, the software application of the AR device may have pre-definedlocations within which the user can position virtual objects, orautomatic or default locations where virtual objects may be positionedby the user.

In some embodiments, the AR device may present a menu option for theuser to move the virtual object to a pre-defined position relative tothe user, for example, 90 degrees to the right of the user's currentfield of view. The hardware and software application of the AR devicemay also be configured to allow the user to move the virtual object to anew position by changing the orientation and/or position of the ARdevice. For example, the AR device may be configured to enter a“position virtual content” mode to allow the user to select a newposition of the virtual object by movements. In the “position virtualcontent” mode, the AR device may track the orientation and/or positionof the AR device for new placement of the virtual object. In this mode,AR device 401 may interpret motions of the head, for example, asproviding a desired relative change in perceived location of the ARcontent and may move the AR content accordingly. In such a mode, the ARcontent may appear to the user as locked in the center of the field ofview while the head rotates and “moves” the AR content to its new,desired location. For example, in FIG. 4, if the user wishes toreposition the AR content 90 degrees to the right, the user could selectthe AR content (e.g., by menu option, voice command, reticlepositioning, activation of a selection tag associated with the ARcontent, input through a hardware-based input device, etc.) forrepositioning. Once selected, the AR device 401 could lock the ARcontent at its current location on the display associated with the ARdevice (e.g., in the center of the display). Then, the user could rotatehis head 90 degrees to the right to position the AR content at the newlocation of (5′, 0′, 5′). After repositioning, the AR device can exitthe repositioning mode and drop the AR content in its new, snappedlocation. As noted previously, this repositioning of AR content can alsobe accomplished using predetermined presets rather than movements of theuser's head.

After snapping the AR content in the repositioned location, the ARdevice may return to a “viewing mode” in which the AR content appears toremain fixed in coordinate system x′, y′, z′. In viewing mode, rotationsof the user's head will result in a changing direction of field of viewvector, v, which may result in a changing real world view. The locationof the AR content, however, will appear to remain fixed in itsrepositioned location. Therefore, rotation of the user's head in viewingmode will, in most embodiments, not result in a corresponding motion ofthe AR content. Instead, this content will appear to reside at itsrepositioned location, and the user may access it by looking in thedirection of the repositioned location.

FIG. 5 is a diagrammatic representation of snapped AR content 500consistent with the exemplary disclosed embodiments. In this example, ARcontent 502 is snapped 90 degrees to the right of the user's field ofview, from the initial position of (0′, 5′, 5′) shown in FIG. 4, tocoordinates of (5′,0′, 5′) according to the second coordinate system.Here the user's field of view again may be defined by a vector v. Asshown in FIG. 5, the vector v has an origin at the position of the ARdevice 501 which is defined to be (0, 0, 5) in the first coordinatesystem, and proceeds along the y-axis. As a result of the AR contentsnapping, the AR content 502 may be moved outside of the user's field ofview in this example. Whether the AR content is within or outside of theuser's field of view, however, may depend on the capabilities of theparticular display device and the viewing angle associated with thefield of view of the display. It should be understood that the ARcontent may be snapped to be within the user's field of view consistentwith the disclosed embodiments.

As shown in FIG. 5, the snapped AR content 502 is moved 90 degrees (inthis exemplary embodiment) to the right of the user's field of view byuser input. In some embodiments, the snapped AR content 502 may beredisplayed in the user's field of view when the user changes theorientation of the AR device during viewing mode, for example, by movingthe AR device 90 degree right such that vector v intersects AR content502. In other words, the snapped AR content may be redisplayed withinthe user's field of view in response to the movement of the orientationand/or position of the AR device.

In some embodiments, the software application of the AR device maypresent an option to the user to enter an AR content “locked mode” wherethe AR content will appear to remain at its repositioned locationrelative to a certain reference user view vector, v. In locked mode,movements affecting the position of AR device 501 relative to coordinatesystem x, y, z may be duplicated in rendering the AR content 502. Thatis, the software application of the AR device may associate the twocoordinate systems in such a way that any translations, rotations, orchanges to the position of the AR device or the field of view in thefirst coordinate system are duplicated by similar translations,rotations, or changes to the position of the virtual object in thesecond coordinate system. This locked mode may allow the user to movefreely within an environment without modifying or losing the relativeposition of the snapped AR object. For example, if the user positions anAR object at a location 45 degrees to the left of the user's field ofview center (e.g., to enable access to the AR content with a 45 degreeleftward rotation of the head during viewing mode), then the user maywish to retain that 45 degree relative relationship between the field ofview vector, v, and the position of the AR content even if the usermoves to an entirely different working area where the user's field ofview vector points in a direction than the vector used as a reference toestablish the relative position of the AR content. In some respects,“locked mode” allows the user to “carry” the AR content to a new userlocation while maintaining the desired relative positioning of the ARcontent. Upon reaching the new location, the user can exit lock mode,and the new direction of user view vector, v, may be used as thereference for the relative position of the AR content. In this way, theselected position of the AR content may be maintained in a desiredlocation despite translation or rotational movements associated with achange in user position.

In some embodiments, the software application of the AR device mayenable entry into the locked mode through selection of a menu item(e.g., “Lock Snapped Content”) to the user. The user may toggle theoption of “Lock Snapped Content” to be in an on or off state. When the“View Snapped Content” option is selected to be on, the position of theAR content may track the position of the AR device, and anytranslations, rotations, or changes to the position of the AR device orthe field of view in the first coordinate system may be duplicated asidentical translations, rotations, or changes to the perceived positionof the virtual AR content in the second coordinate system. On the otherhand, when the “Lock Snapped Content” option is turned off, the positionof the AR content may be independent from the movement of the AR device,and may not track the position of the AR device.

FIG. 6 illustrates an example 600 of AR content movement consistent withthe exemplary disclosed embodiments. As shown in FIG. 6, the AR device601 is initially oriented toward the direction of the y-axis, and thecenter of the user's field of view may be defined as a vector v₁. Thevector v₁ has an origin at coordinates of (0, 0, 5) in the firstcoordinate system, and proceeds along the y-axis. The initialcoordinates of the AR content 602 may be defined to be (0′, 5′, 5′) inthe second coordinate system. To reposition the AR content, the user mayplace AR device 601 in locked mode such that movements of AR device 601cause similar apparent movements in the virtual location of AR content602. For example, to reposition AR content 90 degrees to the right, theuser may select locked mode and then rotate the AR device 90 degree tothe right of the original field of view vector. Correspondingly, theuser's field of view is changed to a new vector v₂. The vector v₂ has anorigin at coordinates of (0, 0, 5) in the first coordinate system, andproceeds along the x-axis. In locked mode, the changes to the positionof the AR device 601 or the field of view in the first coordinate systemmay be duplicated by similar changes to the position of the AR content602 in the second coordinate system. As a result, the AR content 602also appears to be rotated 90 degrees to the right of vector v₁ in thesecond coordinate system, and the perceived position of the AR content602 may change to (5′, 0′, 5′) relative to the second coordinate system.In doing so, the AR content 602 is locked to be within the user'scurrent field of view regardless of the movement of AR device.

Once the AR content has been located to a desired position, the user mayexit locked mode and re-enter viewing mode. In viewing mode, movementsof AR device 601 may not affect the perceived location of AR content602. Rather, in the example above, the AR content would appear to remainat position (5′, 0′, 5′) regardless of rotation of the AR device 601.Such a viewing mode enables a user to look along vector v₁ until theuser desires to access the AR content. To access the AR content in itsnew location, the user would rotate his head thereby turning AR device601 such that the field of view vector would correspond with vector v₂shown in FIG. 6. At this point, vector v₂ would intersect with theperceived location of AR content 602 and the software executed by theprocessor device of system 120/AR device 601 would render the AR contentin the center of the user's field of view to create the perceived effectthat the user is looking at the AR content as a position of (5′, 0′, 5′)relative the second coordinate system.

It should be noted that the described viewing mode may include atranslation lock component. In translation-locked viewing mode, theperceived location of AR content 602 may remain unchanged as the ARdevice 601 rotates (which enables the user to “look” toward and awayfrom the AR content). Translation of the AR device, however, may resultin similar translation of the perceived location of the AR content. Forexample, if the AR content is positioned 90 degrees to the right of areference field of view vector, then in translation-locked mode, walkingwith AR device 601 may result in a perception that AR content similarlytranslating through space while hovering at a position 90 degrees to theright of the user's current center of field of view vector.

FIG. 7 illustrates an example 700 of AR content movement consistent withthe exemplary disclosed embodiments. Specifically, FIG. 7 provides anexample of translation-locked viewing mode. In this example, AR device701 may be initially oriented around a vector v₁ facing in the negativey direction and having an origin at coordinates (0, 10, 5). AR content702 may be positioned at a perceived location of (0′, 0′, 5′). In thetranslation-locked viewing mode, if the user moves toward the originalong the negative y direction (e.g., such that the field of view vectorv₁ having an origin at (0, 10, 5) translates to a field of view vectorv₂ having an origin at (0, 5, 5)), then the display of device 701 willbe correspondingly updated such that AR content 702 appears to translatealong with the user's movements. Thus, in this mode, rotations of theuser's head and corresponding rotations of the AR device 701 may resultin corresponding motion of the displayed AR content within the user'sfield of view (e.g., the user may be able to freely look at and awayfrom the AR content by rotating the head), but no apparent movement inthe perceived location of the AR content relative to coordinate systemx, y, z. On the other hand, in translation-locked viewing mode,translations of AR device 701 may result in changes in the perceivedlocation of AR content 702 relative to coordinate system x, y, z.

FIG. 8 illustrates an example 800 of updating a display of AR contentconsistent with the exemplary disclosed embodiments. Particularly, FIG.8 provides an example of normal viewing mode where a user can access ARcontent located at a perceived location relative to coordinate systemx′, y′, z′ by looking in the direction of that perceived location. Asshown in FIG. 8, the AR device 801 is initially oriented toward thedirection of the y-axis, and the user's field of view is centered abouta vector The vector v₁ has an origin at coordinates of (0, 0, 5) in thefirst coordinate system, and proceeds along the positive y-axis. Theinitial coordinates of the AR content 802 are defined to be (5′, 0′, 5′)in the second coordinate system. The user then rotates the AR device 90degrees to the right. Correspondingly, the user's field of view ischanged to be centered about a vector v₂. The vector v₂ has an origin atcoordinates of (0, 0, 5) in the first coordinate system, and proceedsalong the positive x-axis. In this viewing mode, the position of the ARcontent 802 in the second coordinate system does not track therotational changes of the position of the AR device 801 relative to thefirst coordinate system. Accordingly, the AR content 802 appears in thedisplay of device 801 to remain at the perceived position of (5′, 0′,5′) in the second coordinate system

In some implementations, the user may select an option in the AR devicesoftware application that disassociates the two coordinate systems fromeach other such that any translations, rotations, or changes to theposition of the AR device or the field of view in the first coordinatesystem are performed independently of the second coordinate system. Forexample, in some embodiments, the user may choose to snap the AR contentat a particular location relative to the second coordinate system (x, y,z) and effectively “leave it” in that location. This embodiment may bedescribed as a real-world locked viewing mode. For example, whenpresented with AR content, the user may wish to snap that content at afixed position in the second coordinate system, for example, at adesired perceived location relative to a real world object. Then, theuser may move away from the real world object to a new location. Lookingback toward the AR content, the AR content will be rendered on theuser's display such that the user perceives the AR content as remainingin the selected position. That is, the AR content will appear fartheraway if the user moves away from the selected location.

Such AR content may also be made available to other users of system 100.For example, if AR content is provided to one or more users, and any ofthose users snaps the content in a selected position relative tocoordinate system x′, y′, z′, then other users of system 100 mayperceive the same AR content in the position designated by theposition-selecting user.

FIG. 9 illustrates another example 900 of updating display of AR contentconsistent with the exemplary disclosed embodiments. Specifically, FIG.9 provides an example of real-world locked viewing mode. In thisexample, AR device 901 is initially oriented along the negative y-axisdirection, and the user's field of view is centered about a vector v₁.The vector v₁ has an origin at coordinates of (0, 10, 5) in the firstcoordinate system. The initial coordinates of the perceived location ofAR content 902 are defined to be (0′, 0′, 5′) relative to the secondcoordinate system. The user then moves toward the origin in the negativey direction such that the user's field of view vector translates tovector v₂, which has an origin at coordinates of (0, 5, 5) in the firstcoordinate system and proceeds along the negative y direction. Becausein this mode the AR content is snapped to a real world location, theperceived relationship between the perceived location of the AR contentand the location of the AR device 901 is not maintained. Rather, as ARdevice 901 translates through space, the perceived location of ARcontent 902 will appear to remain at the position of (0′, 0′, 5′) in thesecond coordinate system. In doing so, the perceived relative positionbetween the AR content 902 and the AR device 901 changes, and the ARcontent 902 is displayed on the display of AR device 901 such that itappears nearer to the AR device in the current user's field of view.

When the user initiates the steps to “snap” a virtual object to a newposition, the AR device may set up a first coordinate system for the ARdevice that includes the field of view. The AR content to be snapped maybe stored in memory until it is assigned a location in a secondcoordinate system. The second coordinate system has an origin that maybe defined by the software application of the AR device. The virtualobject may then be snapped to a location, position, orientation definedin the second coordinate system.

In one embodiment, the second coordinate system may be associated withexternal coordinate data that corresponds to locations in the realphysical world (e.g., latitude, longitude and altitude, or indoorlocations defined by a model, map or other means). In anotherembodiment, the first coordinate system may be associated with externalcoordinate data. In yet another embodiment, both coordinate systems maybe associated with external coordinate data.

The sensors in the AR device may provide output signals and/or data tothe software application of the AR device indicative of the currentlocation, orientation and/or bearing of the display unit (and thereforethe user). This information, in turn, may be used to update or changeaspects of images or information presented to the user within the user'sfield of view on the display unit.

In some embodiments, perceived locations of AR content (e.g., relativeto the second coordinate system) may be defined based on recognizedglyphs. Such glyphs may include any object or thing appearing in thereal world that may be recognized within a scene and with which ARcontent may be associated. For example, glyphs may include but are notlimited to images, markers, physical objects, one dimensional or twodimensional codes (e.g., a QR code), people, places, buildings, etc.which may be captured by the AR device (e.g., through an imageacquisition device such as a camera) and recognized by the softwareapplication of the AR device (e.g., through computer vision algorithms).As a result of recognition of a glyph, AR content may be recalled frommemory (either local or from a server system, etc.) and provided to theuser. For example, in some embodiments, the AR content may be renderedon the display of the AR device such that it appears to the user aslocated at a predetermined position relative to the glyph. For example,if the glyph is a QR code on an object, the AR content may be providedsuch that it appears to the user as hovering 6 inches above the object.Of course, and suitable relative location may be used for providing theAR content with respect to the recognized object/glyph. This process maybe referred to as glyph snapping. Notably, glyph-snapped AR content maybe made available to multiple users of system 100 and to all users, nomatter their location, may appear in the same predetermined locationrelative to the recognized glyph.

The second coordinate system may be defined by software application ofthe AR device when a glyph is identified by the hardware and softwareapplication of the AR device. The origin of the second coordinate systemmay correspond to a point rendered on the display of the AR device inrecognition of the glyph. For example, the origin point rendered on thedisplay may be an overlay on top of the glyph, offset from the glyph orin some other location on the user's current field of view.

In some embodiments, the user may select a virtual object, using inputmethods supported by the hardware and software application of the ARdevice. The user may take action to define a new position for thevirtual object in the second coordinate system. The virtual object maybe associated with glyph. As the user and the AR device move and theglyph moves in real-world physical space, the virtual object may berendered on the display in the user's field of view when the glyph isrecognized by the hardware and software applications of the AR device.

In some embodiments, the AR device may be configured to be capable ofperform cardinal snapping, glyph snapping, or a combination of them, forAR content. The AR device may be configured to switch from one snappingmode, such as global snapping, to another snapping mode, such ascardinal snapping, by user input such as menu options, speech command,etc.

It should be further understood that arrangements described herein arefor purposes of example only. As such, those skilled in the art willappreciate that other arrangements and other elements (e.g. machines,interfaces, functions, orders, and groupings of functions, etc.) can beused instead, and some elements may be omitted altogether according tothe desired results. Further, many of the elements that are describedare functional entities that may be implemented as discrete ordistributed components or in conjunction with other components, in anysuitable combination and location.

The present disclosure is not to be limited in terms of the particularembodiments described in this application, which are intended asillustrations of various aspects. Many modifications and variations canbe made without departing from its spirit and scope, as will be apparentto those skilled in the art. Functionally equivalent methods andapparatuses within the scope of the disclosure, in addition to thoseenumerated herein, will be apparent to those skilled in the art from theforegoing descriptions. Such modifications and variations are intendedto fall within the scope of the appended claims.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims, along with the full scope ofequivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

What is claimed is:
 1. An augmented reality (AR) device, comprising: adisplay configured to display real-world content overlayed with ARcontent; at least one sensor configured to provide an output indicativeof an orientation, location, or motion of the AR device; and a processordevice configured to: cause the AR content to be shown on the display ata user-selectable initial location on the display; determine a change inorientation of the AR device based on the output of the at least onesensor; and change a position of the AR content on the display to asecond location on the display, wherein the change in position of the ARcontent from the initial location to the second location is related tothe determined change in orientation of the AR device.
 2. The device ofclaim 1, wherein the change in position of the AR content on the displayis configured to cause a user to perceive the AR content as residing ina fixed location relative to the real-world content.
 3. The device ofclaim 1, wherein the initial location on the display is located awayfrom a center of a field of view of the AR device, the second locationis closer to the center of the field of view of the AR device than theinitial location, and the change in position of the AR content from theinitial location to the second location is caused by rotation of the ARdevice in the direction of a perceived location of the AR content. 4.The device of claim 1, wherein the processor device is configuredreceive information relating to a user selection of the initial locationof the AR content.
 5. The device of claim 4, where selection of theinitial location is facilitated by a user input device.
 6. The device ofclaim 5, wherein the user input device includes one or more of a mouse,pointer device, gesture recognition unit, push button device, touchsensitive surface, or voice recognition unit.
 7. The device of claim 1,wherein the processor device is configured to change the position of theAR content on the display in response to a rotational orientation changeof the AR device.
 8. The device of claim 1, wherein the processor deviceis configured to change the perceived position of the AR content aspresented on the display in response to both a rotational and atranslational orientation change of the AR device.
 9. The device ofclaim 1, wherein causing the AR content to be shown is in response to arecognized glyph.
 10. The device of claim 9, wherein the AR content sshown on the display such that the AR content resides at a predeterminedperceived location relative to the recognized glyph.
 11. The device ofclaim 1, further including a data interface configured to receiveinformation associated with the AR content from a location remote fromthe AR device.
 12. The device of claim 11, wherein the data interfaceincludes a data connection to the Internet.
 13. The device of claim 11,wherein the data interface includes a wireless transceiver device. 14.The device of claim 1, wherein the processor device is furtherconfigured to snap the AR content to a location in physical space tomake the AR content accessible to other users in proximity to thelocation in physical space.
 15. A method of displaying augmented reality(AR) content comprising: causing the AR content to be shown at auser-selectable initial location on a display device associated with anAR device; determining a change in orientation of the AR device based onan output of at least one sensor associated with the AR device; andchanging a position of the AR content on the display to a secondlocation on the display, wherein the change in position of the ARcontent from the initial location to the second location is related tothe determined change in orientation of the AR device.